Papers with large language model inference
Motivating Next-Gen Accelerators with Flexible N:M Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches (2026.acl-industry)
Copied to clipboard
Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina, Vladimir Smirnov, Redko Dmitry, Alexey Dontsov, Maxim Zhelnin, Evgeny Burnaev, Egor Shvetsov
| Challenge: | Recent studies show that sparsification is not supported in large language models. |
| Approach: | They propose to use activation pruning to accelerate large language models with sparsification . they compare activation pruners with weight pruner and activater pruning with activation . |
| Outcome: | The proposed approach outperforms weight pruning at matched sparsity levels. |
Jakiro: Boosting Speculative Decoding via Decoupled MoE (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to accelerate large language model inference have a fundamental limitation: candidates at the same tree layer share identical feature representations, constraining diversity and diminishing overall effectiveness. |
| Approach: | They propose a decoupled mixture of experts (MoE) into a draft model to generate diverse tokens from distinct feature spaces. |
| Outcome: | The proposed approach achieves significant speedups over strong baselines, with notable improvements in non-greedy scenarios where token diversity is crucial. |
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion (2025.naacl-long)
Copied to clipboard
Jacob K Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, Ferdinando Fioretto
| Challenge: | Existing methods to accelerate large language model inference are limited by the reliance on incremental token generation in existing draft models. |
| Approach: | They propose an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences and allows parallelization of both the drafting and verification steps. |
| Outcome: | The proposed approach provides 7.2x speedups over standard generation processes and 1.75x speed ups over existing speculative decoding approaches. |
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding is limited by sequential stage dependencies that prevent full hardware utilization. |
| Approach: | They propose a framework that generalizes speculative decoding to use multiple models arranged in a hierarchical pipeline and enables asynchronous execution with lightweight coordination for prediction verification and rollback. |
| Outcome: | The proposed framework achieves 2.25 tokens/unit through pipelined parallelism with multiple models arranged in a hierarchical pipeline. |
From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding (SD) allows a lightweight draft model to propose outputs that a stronger target model verifies. |
| Approach: | They propose a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. |
| Outcome: | Experiments show that SpecGuard outperforms both SD and reward-guided SD in accuracy and reliability tests. |
Speculative Sampling via Exponential Races (2025.findings-acl)
Copied to clipboard
| Challenge: | Speculative decoding accelerates large language model inference using a smaller draft model. |
| Approach: | They propose a speculative decoding method that generates multiple draft tokens for each model evaluation using a more efficient draft model. |
| Outcome: | The proposed method matches state-of-the-art performance and is based on exponential races. |
EDSD: Entropy-Driven Design for Faster Speculative Decoding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for speculative decoding incur substantial training overhead to mitigate information misalignment between autoregressive draft model training and decoding. |
| Approach: | They propose an Entropy-Driven Speculative Decoding framework that uses entropy as a unified, interpretable signal for both draft model training and architectural design. |
| Outcome: | Experiments on seven large language models show that EDSD improves training efficiency by 24.8% and increases acceptance length by 4.0% compared to state-of-the-art methods. |